{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hybrid Search\n",
    "\n",
    "Qdrant supports hybrid search by combining search results from sparse and dense vectors.\n",
    "\n",
    "dense vectors are the ones you have probably already been using -- embedding models from OpenAI, BGE, SentenceTransformers, etc. are typically dense embedding models. They create a numerical representation of a piece of text, represented as a long list of numbers. These dense vectors can capture rich semantics across the entire piece of text.\n",
    "\n",
    "sparse vectors are slightly different. They use a specialized approach or model (TF-IDF, BM25, SPLADE, etc.) for generating vectors. These vectors are typically mostly zeros, making them sparse vectors. These sparse vectors are great at capturing specific keywords and similar small details.\n",
    "\n",
    "This notebook walks through setting up and customizing hybrid search with Qdrant and \"prithvida/Splade_PP_en_v1\" variants from Huggingface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install llama-index\n",
    "# !pip install llama-index-vector-stores-qdrant llama-index-readers-file llama-index-embeddings-fastembed llama-index-llms-openai\n",
    "# !pip install -U qdrant_client fastembed\n",
    "# !pip install python-dotenv\n",
    "# !pip install ragas\n",
    "# !pip install trulens_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d1d4c0f5aef45b690573db3d2970c4a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 5 files:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "import os\n",
    "from dotenv import load_dotenv\n",
    "from IPython.display import Markdown, display\n",
    "\n",
    "# qdrant official client\n",
    "from qdrant_client import QdrantClient, AsyncQdrantClient\n",
    "\n",
    "# LLama-index dependencies\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "from llama_index.vector_stores.qdrant import QdrantVectorStore\n",
    "from llama_index.embeddings.fastembed import FastEmbedEmbedding\n",
    "from llama_index.core import Settings\n",
    "\n",
    "# setting the embedding model to BAAI/bge-base-en-v1.5 and FastEmbed to inference these models\n",
    "# Settings.embed_model = FastEmbedEmbedding(model_name=\"BAAI/bge-base-en-v1.5\")\n",
    "Settings.embed_model = FastEmbedEmbedding(model_name=\"BAAI/bge-base-en-v1.5\")\n",
    "# embed_model = FastEmbedEmbedding(model_name=\"BAAI/bge-base-en-v1.5\" , max_length=1024)\n",
    "\n",
    "# load all environment variables\n",
    "load_dotenv()\n",
    "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")\n",
    "QDRANT_CLOUD_ENDPOINT = os.getenv(\"QDRANT_CLOUD_ENDPOINT\")\n",
    "QDRANT_API_KEY = os.getenv(\"QDRANT_API_KEY\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading files: 100%|██████████| 1/1 [00:00<00:00,  5.59file/s]\n"
     ]
    }
   ],
   "source": [
    "# lets loading the documents using SimpleDirectoryReader\n",
    "from llama_index.core import Document\n",
    "reader = SimpleDirectoryReader(\"./data/69_markdown_test/\" , recursive=True)\n",
    "documents = reader.load_data(show_progress=True)\n",
    "\n",
    "# combining all the documents into a single document for later chunking and splitting\n",
    "# documents = Document(text=\"\\n\\n\".join([doc.text for doc in documents]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up Vector Database\n",
    "\n",
    "We will be using qDrant as the Vector database\n",
    "There are 4 ways to initialize qdrant \n",
    "\n",
    "1. Inmemory\n",
    "```python\n",
    "client = qdrant_client.QdrantClient(location=\":memory:\")\n",
    "```\n",
    "2. Disk\n",
    "```python\n",
    "client = qdrant_client.QdrantClient(path=\"./data\")\n",
    "```\n",
    "3. Self hosted or Docker\n",
    "```python\n",
    "\n",
    "client = qdrant_client.QdrantClient(\n",
    "    # url=\"http://<host>:<port>\"\n",
    "    host=\"localhost\",port=6333\n",
    ")\n",
    "```\n",
    "\n",
    "4. Qdrant cloud\n",
    "```python\n",
    "client = qdrant_client.QdrantClient(\n",
    "    url=QDRANT_CLOUD_ENDPOINT,\n",
    "    api_key=QDRANT_API_KEY,\n",
    ")\n",
    "```\n",
    "\n",
    "for this notebook we will be using qdrant cloud"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Both client and aclient are provided. If using `:memory:` mode, the data between clients is not synced.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7e7bbfcbdc3d4f97b6b9325dc0937317",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 5 files:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "adeba327379d4c95b3cb073f172aebe6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Fetching 5 files:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# creating a qdrant client instance\n",
    "\n",
    "client = QdrantClient(\n",
    "    # you can use :memory: mode for fast and light-weight experiments,\n",
    "    # it does not require to have Qdrant deployed anywhere\n",
    "    # but requires qdrant-client >= 1.1.1\n",
    "    # location=\":memory:\"\n",
    "    # otherwise set Qdrant instance address with:\n",
    "    url=QDRANT_CLOUD_ENDPOINT,\n",
    "    # otherwise set Qdrant instance with host and port:\n",
    "    # host=\"localhost\",\n",
    "    # port=6333\n",
    "    # set API KEY for Qdrant Cloud\n",
    "    api_key=QDRANT_API_KEY,\n",
    "    # path=\"./db/\"\n",
    ")\n",
    "\n",
    "# setting up asynchronous client\n",
    "aclient = AsyncQdrantClient(\n",
    "    # you can use :memory: mode for fast and light-weight experiments,\n",
    "    # it does not require to have Qdrant deployed anywhere\n",
    "    # but requires qdrant-client >= 1.1.1\n",
    "    # location=\":memory:\"\n",
    "    # otherwise set Qdrant instance address with:\n",
    "    url=QDRANT_CLOUD_ENDPOINT,\n",
    "    # otherwise set Qdrant instance with host and port:\n",
    "    # host=\"localhost\",\n",
    "    # port=6333\n",
    "    # set API KEY for Qdrant Cloud\n",
    "    api_key=QDRANT_API_KEY,\n",
    "    # path=\"./db/\"\n",
    ")\n",
    "\n",
    "Settings.chunk_size = 512\n",
    "\n",
    "## enable Hybrid RAG here\n",
    "vector_store = QdrantVectorStore(client=client, \n",
    "                                 aclient=aclient ,\n",
    "                                 collection_name=\"2_Hybrid_RAG\",\n",
    "                                 enable_hybrid=True , \n",
    "                                 batch_size=20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5037565f949942ef8b00fbf98424598a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Parsing nodes:   0%|          | 0/43 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "097513faa08047b098a5fa918f290217",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/45 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "## ingesting data into vector database\n",
    "\n",
    "## lets set up an ingestion pipeline\n",
    "\n",
    "from llama_index.core.node_parser import TokenTextSplitter\n",
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "from llama_index.core.node_parser import MarkdownNodeParser\n",
    "from llama_index.core.ingestion import IngestionPipeline\n",
    "\n",
    "pipeline = IngestionPipeline(\n",
    "    transformations=[\n",
    "        # MarkdownNodeParser(include_metadata=True),\n",
    "        # TokenTextSplitter(chunk_size=500, chunk_overlap=20),\n",
    "        SentenceSplitter(chunk_size=1024, chunk_overlap=20),\n",
    "        Settings.embed_model,\n",
    "    ],\n",
    "    vector_store=vector_store,\n",
    "\n",
    ")\n",
    "Settings.chunk_size = 512\n",
    "# Ingest directly into a vector db\n",
    "nodes = pipeline.run(documents=documents , show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting Up Retriever"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_vector_store(vector_store=vector_store)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modifying Prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_prompt_str = (\n",
    "    \"Context information is below.\\n\"\n",
    "    \"---------------------\\n\"\n",
    "    \"{context_str}\\n\"\n",
    "    \"---------------------\\n\"\n",
    "    \"Given the context information and not prior knowledge, \"\n",
    "    \"answer the question: {query_str}\\n\"\n",
    ")\n",
    "\n",
    "refine_prompt_str = (\n",
    "    \"We have the opportunity to refine the original answer \"\n",
    "    \"(only if needed) with some more context below.\\n\"\n",
    "    \"------------\\n\"\n",
    "    \"{context_msg}\\n\"\n",
    "    \"------------\\n\"\n",
    "    \"Given the new context, refine the original answer to better \"\n",
    "    \"answer the question: {query_str}. \"\n",
    "    \"If the context isn't useful, output the original answer again.\\n\"\n",
    "    \"Original Answer: {existing_answer}\"\n",
    ")\n",
    "\n",
    "from llama_index.core import ChatPromptTemplate\n",
    "\n",
    "# Text QA Prompt\n",
    "chat_text_qa_msgs = [\n",
    "    (\"system\",\"You are a AI assistant who is well versed with medical information and only answer question per training to the medical domain\"),\n",
    "    (\"user\", qa_prompt_str),\n",
    "]\n",
    "text_qa_template = ChatPromptTemplate.from_messages(chat_text_qa_msgs)\n",
    "\n",
    "# Refine Prompt\n",
    "chat_refine_msgs = [\n",
    "    (\"system\",\"Always answer the question, even if the context isn't helpful.\",),\n",
    "    (\"user\", refine_prompt_str),\n",
    "]\n",
    "refine_template = ChatPromptTemplate.from_messages(chat_refine_msgs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from llama_index.llms.openai import OpenAI\n",
    "llm = OpenAI()\n",
    "\n",
    "RAG = index.as_query_engine(\n",
    "        text_qa_template=text_qa_template,\n",
    "        refine_template=refine_template,\n",
    "        llm=llm,\n",
    "        similarity_top_k=2,\n",
    "        sparse_top_k=12,\n",
    "        vector_store_query_mode=\"hybrid\"\n",
    "        )\n",
    "\n",
    "\n",
    "response = RAG.query(\"Tell me more about Dosage adjustment is required in patients whose creatinine clearance is less than 30 mL/min and who are not receiving regularly scheduled hemodialysis. (8.6) See 17 for PATIENT COUNSELING INFORMATION \")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "In patients with renal impairment whose known creatinine clearance is less than 30 mL/min and who are not receiving regularly scheduled hemodialysis, the recommended two-dose regimen for DALVANCE is 750 mg followed one week later by 375 mg. No dosage adjustment is recommended for patients receiving regularly scheduled hemodialysis, and DALVANCE can be administered without regard to the timing of hemodialysis. This information is based on the details provided in section 8.6 of the document."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(str(response.response)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Performing Evaluation using RAGAS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Synthetic Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "eaa6e09e42b34463898a0dae11b17b2d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "embedding nodes:   0%|          | 0/106 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Filename and doc_id are the same for all nodes.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b8a8d54c5ecd4733ad234f33bfcfbc45",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating:   0%|          | 0/10 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from ragas.testset.generator import TestsetGenerator\n",
    "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
    "\n",
    "# documents = load your documents\n",
    "\n",
    "# generator with openai models\n",
    "generator_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n",
    "critic_llm = ChatOpenAI(model=\"gpt-4\")\n",
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "generator = TestsetGenerator.from_langchain(\n",
    "    generator_llm,\n",
    "    critic_llm,\n",
    "    embeddings\n",
    ")\n",
    "\n",
    "# Change resulting question type distribution\n",
    "distributions = {\n",
    "    simple: 0.5,\n",
    "    multi_context: 0.4,\n",
    "    reasoning: 0.1\n",
    "}\n",
    "\n",
    "# use generator.generate_with_llamaindex_docs if you use llama-index as document loader\n",
    "\n",
    "# the document passes here is from the 2nd cell\n",
    "testset = generator.generate_with_llamaindex_docs(documents, 10, distributions) \n",
    "testset = testset.to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# testset.to_csv('./archives/eval_data3.csv', index=False)\n",
    "import pandas as pd\n",
    "\n",
    "testset = pd.read_csv('./archives/eval_data.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Evaluation using Truelens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "🦑 Tru initialized with db url sqlite:///default.sqlite .\n",
      "🛑 Secret keys may be written to the database. See the `database_redact_keys` option of Tru` to prevent this.\n"
     ]
    }
   ],
   "source": [
    "from trulens_eval import Tru\n",
    "tru = Tru()\n",
    "\n",
    "# tru.reset_database()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ In groundedness_measure_with_cot_reasons, input source will be set to __record__.app.query.rets.source_nodes[:].node.text.collect() .\n",
      "✅ In groundedness_measure_with_cot_reasons, input statement will be set to __record__.main_output or `Select.RecordOutput` .\n",
      "✅ In relevance, input prompt will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In relevance, input response will be set to __record__.main_output or `Select.RecordOutput` .\n",
      "✅ In context_relevance_with_cot_reasons, input question will be set to __record__.main_input or `Select.RecordInput` .\n",
      "✅ In context_relevance_with_cot_reasons, input context will be set to __record__.app.query.rets.source_nodes[:].node.text .\n"
     ]
    }
   ],
   "source": [
    "from trulens_eval.feedback.provider import OpenAI\n",
    "from trulens_eval import Feedback\n",
    "import numpy as np\n",
    "\n",
    "# Initialize provider class\n",
    "provider = OpenAI()\n",
    "\n",
    "# select context to be used in feedback. the location of context is app specific.\n",
    "from trulens_eval.app import App\n",
    "context = App.select_context(RAG)\n",
    "\n",
    "# Define a groundedness feedback function\n",
    "f_groundedness = (\n",
    "    Feedback(provider.groundedness_measure_with_cot_reasons)\n",
    "    .on(context.collect()) # collect context chunks into a list\n",
    "    .on_output()\n",
    ")\n",
    "\n",
    "# Question/answer relevance between overall question and answer.\n",
    "f_answer_relevance = (\n",
    "    Feedback(provider.relevance)\n",
    "    .on_input_output()\n",
    ")\n",
    "# Question/statement relevance between question and each context chunk.\n",
    "f_context_relevance = (\n",
    "    Feedback(provider.context_relevance_with_cot_reasons)\n",
    "    .on_input()\n",
    "    .on(context)\n",
    "    .aggregate(np.mean)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from trulens_eval import TruLlama\n",
    "\n",
    "tru_query_engine_recorder = TruLlama(RAG,app_id=\"2_Hybrid_RAG\",feedbacks=[f_groundedness, f_answer_relevance, f_context_relevance])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4203ae80a9b4457cb8f9a2920da1e307",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73b1a27aaec041c5ae81f040ddd3094c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48a6d440699c4984998c0503f4bab027",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fb8a19e5ef944f7ea08dc3d24beb76dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/5 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Adithya\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f7e9c871a9dd402097df7e5d1a310ed6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2b2b5c69e08d426682a5f6e92398f9eb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1024b265fe2042ca9be164eafb909ed6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Adithya\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a6ef76103de45bd85ed3c0eb0c42d49",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\Adithya\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f918d7e5821e43c48cd4ca1a8d6eb88d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3426d7e3c3b04cb3a874cee98578b986",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Groundedness per statement in source:   0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "eval_questions = testset['question'].to_list()\n",
    "\n",
    "with tru_query_engine_recorder as recording:\n",
    "    for question in eval_questions:\n",
    "        response = RAG.query(question)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "records, feedback = tru.get_records_and_feedback(app_ids=[])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>app_id</th>\n",
       "      <th>app_json</th>\n",
       "      <th>type</th>\n",
       "      <th>record_id</th>\n",
       "      <th>input</th>\n",
       "      <th>output</th>\n",
       "      <th>tags</th>\n",
       "      <th>record_json</th>\n",
       "      <th>cost_json</th>\n",
       "      <th>perf_json</th>\n",
       "      <th>ts</th>\n",
       "      <th>relevance</th>\n",
       "      <th>groundedness_measure_with_cot_reasons</th>\n",
       "      <th>context_relevance_with_cot_reasons</th>\n",
       "      <th>relevance_calls</th>\n",
       "      <th>groundedness_measure_with_cot_reasons_calls</th>\n",
       "      <th>context_relevance_with_cot_reasons_calls</th>\n",
       "      <th>latency</th>\n",
       "      <th>total_tokens</th>\n",
       "      <th>total_cost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0_Naive_RAG</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
       "      <td>record_hash_2a084b36e219af3247b101b8a0c3d7d0</td>\n",
       "      <td>\"What are the Gram-positive microorganisms tha...</td>\n",
       "      <td>\"DALVANCE (dalbavancin) is effective against G...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_2a084b36e219af3247b...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-05-25T15:22:24.419886\", \"...</td>\n",
       "      <td>2024-05-25T15:22:27.628004</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.5</td>\n",
       "      <td>[{'args': {'prompt': 'What are the Gram-positi...</td>\n",
       "      <td>[{'args': {'source': ['15 References\\n\\n1. Cli...</td>\n",
       "      <td>[{'args': {'question': 'What are the Gram-posi...</td>\n",
       "      <td>3</td>\n",
       "      <td>1509</td>\n",
       "      <td>0.002285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0_Naive_RAG</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
       "      <td>record_hash_233e2b854caa1fae1f9a5cb45caa7fae</td>\n",
       "      <td>\"What were the characteristics of the patients...</td>\n",
       "      <td>\"I'm sorry, but the provided context informati...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_233e2b854caa1fae1f9...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-05-25T15:22:28.229274\", \"...</td>\n",
       "      <td>2024-05-25T15:22:31.224825</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.5</td>\n",
       "      <td>[{'args': {'prompt': 'What were the characteri...</td>\n",
       "      <td>[{'args': {'source': ['Specific Populations\\n\\...</td>\n",
       "      <td>[{'args': {'question': 'What were the characte...</td>\n",
       "      <td>2</td>\n",
       "      <td>1959</td>\n",
       "      <td>0.002969</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0_Naive_RAG</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
       "      <td>record_hash_253dcd210a269b21a054ad3d853719e0</td>\n",
       "      <td>\"How do CYP450 substrates interact with dalbav...</td>\n",
       "      <td>\"In vitro studies have shown that dalbavancin ...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_253dcd210a269b21a05...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-05-25T15:22:31.783235\", \"...</td>\n",
       "      <td>2024-05-25T15:22:34.618613</td>\n",
       "      <td>0.9</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>0.8</td>\n",
       "      <td>[{'args': {'prompt': 'How do CYP450 substrates...</td>\n",
       "      <td>[{'args': {'source': ['12.3 Pharmacokinetics\\n...</td>\n",
       "      <td>[{'args': {'question': 'How do CYP450 substrat...</td>\n",
       "      <td>2</td>\n",
       "      <td>1274</td>\n",
       "      <td>0.001941</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0_Naive_RAG</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
       "      <td>record_hash_798023af5ef331a3e6846ff7d41e5a16</td>\n",
       "      <td>\"What are the warnings and precautions regardi...</td>\n",
       "      <td>\"The warnings and precautions regarding hypers...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_798023af5ef331a3e68...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-05-25T15:22:35.088900\", \"...</td>\n",
       "      <td>2024-05-25T15:22:39.301764</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.983333</td>\n",
       "      <td>0.5</td>\n",
       "      <td>[{'args': {'prompt': 'What are the warnings an...</td>\n",
       "      <td>[{'args': {'source': ['3 Dosage Forms And Stre...</td>\n",
       "      <td>[{'args': {'question': 'What are the warnings ...</td>\n",
       "      <td>4</td>\n",
       "      <td>2147</td>\n",
       "      <td>0.003285</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0_Naive_RAG</td>\n",
       "      <td>{\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...</td>\n",
       "      <td>RetrieverQueryEngine(llama_index.core.query_en...</td>\n",
       "      <td>record_hash_5dacdded6fdd9bf4f53eca8885b3dc39</td>\n",
       "      <td>\"What is the recommended dosage regimen for DA...</td>\n",
       "      <td>\"The recommended dosage regimen for DALVANCE f...</td>\n",
       "      <td>-</td>\n",
       "      <td>{\"record_id\": \"record_hash_5dacdded6fdd9bf4f53...</td>\n",
       "      <td>{\"n_requests\": 1, \"n_successful_requests\": 1, ...</td>\n",
       "      <td>{\"start_time\": \"2024-05-25T15:22:39.807016\", \"...</td>\n",
       "      <td>2024-05-25T15:22:42.558752</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.0</td>\n",
       "      <td>[{'args': {'prompt': 'What is the recommended ...</td>\n",
       "      <td>[{'args': {'source': ['Full Prescribing Inform...</td>\n",
       "      <td>[{'args': {'question': 'What is the recommende...</td>\n",
       "      <td>2</td>\n",
       "      <td>2091</td>\n",
       "      <td>0.003166</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        app_id                                           app_json  \\\n",
       "0  0_Naive_RAG  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "1  0_Naive_RAG  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "2  0_Naive_RAG  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "3  0_Naive_RAG  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "4  0_Naive_RAG  {\"tru_class_info\": {\"name\": \"TruLlama\", \"modul...   \n",
       "\n",
       "                                                type  \\\n",
       "0  RetrieverQueryEngine(llama_index.core.query_en...   \n",
       "1  RetrieverQueryEngine(llama_index.core.query_en...   \n",
       "2  RetrieverQueryEngine(llama_index.core.query_en...   \n",
       "3  RetrieverQueryEngine(llama_index.core.query_en...   \n",
       "4  RetrieverQueryEngine(llama_index.core.query_en...   \n",
       "\n",
       "                                      record_id  \\\n",
       "0  record_hash_2a084b36e219af3247b101b8a0c3d7d0   \n",
       "1  record_hash_233e2b854caa1fae1f9a5cb45caa7fae   \n",
       "2  record_hash_253dcd210a269b21a054ad3d853719e0   \n",
       "3  record_hash_798023af5ef331a3e6846ff7d41e5a16   \n",
       "4  record_hash_5dacdded6fdd9bf4f53eca8885b3dc39   \n",
       "\n",
       "                                               input  \\\n",
       "0  \"What are the Gram-positive microorganisms tha...   \n",
       "1  \"What were the characteristics of the patients...   \n",
       "2  \"How do CYP450 substrates interact with dalbav...   \n",
       "3  \"What are the warnings and precautions regardi...   \n",
       "4  \"What is the recommended dosage regimen for DA...   \n",
       "\n",
       "                                              output tags  \\\n",
       "0  \"DALVANCE (dalbavancin) is effective against G...    -   \n",
       "1  \"I'm sorry, but the provided context informati...    -   \n",
       "2  \"In vitro studies have shown that dalbavancin ...    -   \n",
       "3  \"The warnings and precautions regarding hypers...    -   \n",
       "4  \"The recommended dosage regimen for DALVANCE f...    -   \n",
       "\n",
       "                                         record_json  \\\n",
       "0  {\"record_id\": \"record_hash_2a084b36e219af3247b...   \n",
       "1  {\"record_id\": \"record_hash_233e2b854caa1fae1f9...   \n",
       "2  {\"record_id\": \"record_hash_253dcd210a269b21a05...   \n",
       "3  {\"record_id\": \"record_hash_798023af5ef331a3e68...   \n",
       "4  {\"record_id\": \"record_hash_5dacdded6fdd9bf4f53...   \n",
       "\n",
       "                                           cost_json  \\\n",
       "0  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "1  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "2  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "3  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "4  {\"n_requests\": 1, \"n_successful_requests\": 1, ...   \n",
       "\n",
       "                                           perf_json  \\\n",
       "0  {\"start_time\": \"2024-05-25T15:22:24.419886\", \"...   \n",
       "1  {\"start_time\": \"2024-05-25T15:22:28.229274\", \"...   \n",
       "2  {\"start_time\": \"2024-05-25T15:22:31.783235\", \"...   \n",
       "3  {\"start_time\": \"2024-05-25T15:22:35.088900\", \"...   \n",
       "4  {\"start_time\": \"2024-05-25T15:22:39.807016\", \"...   \n",
       "\n",
       "                           ts  relevance  \\\n",
       "0  2024-05-25T15:22:27.628004        0.9   \n",
       "1  2024-05-25T15:22:31.224825        1.0   \n",
       "2  2024-05-25T15:22:34.618613        0.9   \n",
       "3  2024-05-25T15:22:39.301764        0.8   \n",
       "4  2024-05-25T15:22:42.558752        1.0   \n",
       "\n",
       "   groundedness_measure_with_cot_reasons  context_relevance_with_cot_reasons  \\\n",
       "0                               0.000000                                 0.5   \n",
       "1                               1.000000                                 0.5   \n",
       "2                               0.500000                                 0.8   \n",
       "3                               0.983333                                 0.5   \n",
       "4                               1.000000                                 1.0   \n",
       "\n",
       "                                     relevance_calls  \\\n",
       "0  [{'args': {'prompt': 'What are the Gram-positi...   \n",
       "1  [{'args': {'prompt': 'What were the characteri...   \n",
       "2  [{'args': {'prompt': 'How do CYP450 substrates...   \n",
       "3  [{'args': {'prompt': 'What are the warnings an...   \n",
       "4  [{'args': {'prompt': 'What is the recommended ...   \n",
       "\n",
       "         groundedness_measure_with_cot_reasons_calls  \\\n",
       "0  [{'args': {'source': ['15 References\\n\\n1. Cli...   \n",
       "1  [{'args': {'source': ['Specific Populations\\n\\...   \n",
       "2  [{'args': {'source': ['12.3 Pharmacokinetics\\n...   \n",
       "3  [{'args': {'source': ['3 Dosage Forms And Stre...   \n",
       "4  [{'args': {'source': ['Full Prescribing Inform...   \n",
       "\n",
       "            context_relevance_with_cot_reasons_calls  latency  total_tokens  \\\n",
       "0  [{'args': {'question': 'What are the Gram-posi...        3          1509   \n",
       "1  [{'args': {'question': 'What were the characte...        2          1959   \n",
       "2  [{'args': {'question': 'How do CYP450 substrat...        2          1274   \n",
       "3  [{'args': {'question': 'What are the warnings ...        4          2147   \n",
       "4  [{'args': {'question': 'What is the recommende...        2          2091   \n",
       "\n",
       "   total_cost  \n",
       "0    0.002285  \n",
       "1    0.002969  \n",
       "2    0.001941  \n",
       "3    0.003285  \n",
       "4    0.003166  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "records.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting dashboard ...\n",
      "Config file already exists. Skipping writing process.\n",
      "Credentials file already exists. Skipping writing process.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d351f69d264343e9a270e13909f101df",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Accordion(children=(VBox(children=(VBox(children=(Label(value='STDOUT'), Output())), VBox(children=(Label(valu…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dashboard started at http://192.168.1.5:8501 .\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Popen: returncode: None args: ['streamlit', 'run', '--server.headless=True'...>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tru.run_dashboard()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tru.stop_dashboard()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llm-venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
